Field of the Invention
The present invention relates to an image processing technology for applying image restoration processing to deteriorated image generated by an image pickup apparatus.
Description of the Related Art
A rolling shutter is an electronic shutter system for use with an image pickup apparatus, such as a digital camera, that has an image sensor, such as a CMOS sensor. The rolling shutter sequentially controls a start and end of exposure for each pixel row on an image sensor from a pixel row of one end (front pixel row) to a pixel row of the other end (last pixel row). In the pixel row at which signal reading is completed for one frame, electric charging (exposure) is immediately started for the next frame. Thus, when the signal reading for each pixel row and the next electric charging are continuously repeated, the exposure start time and exposure end time shift between the front pixel row and the last pixel row. In other words, the exposure timing shifts. This shift causes a distortion (rolling shutter distortion) in a captured image in capturing a moving object.
Japanese Patent Laid-Open No. (“JP”) 2012-175621 discloses an image processing method for restraining the rolling shutter distortion in the captured image, which would be otherwise caused by the operation of the rolling shutter on the image sensor that provides the rolling shutter operation.
A focal-plane shutter is a mechanical shutter. The focal-plane shutter controls the exposure of the image sensor by moving a preceding light shield (front curtain) and a following light shield (back curtain) parallel to the image plane of the image sensor. Moreover, instead of the mechanical front curtain of the focal-plane shutter, an electronic front curtain can be used to sequentially reset stored electric charges in the image sensor for each pixel row. When the electronic front curtain is used, only a mechanical back curtain is actually driven, and the exposure time periods for all pixel rows can be maintained constant by controlling the resetting timing of electric chargers for each pixel row in the image sensor in accordance with the driving characteristic of the mechanical back curtain. Nevertheless, the electronic front curtain on the image sensor and the mechanical back curtain are distant from each other in the optical axis direction, and thus a light shield position changes on the image sensor by the mechanical back curtain due to influence of the focal length and exit pupil distance of the image pickup optical system, etc. Thus, uneven brightness in a captured image or an uneven exposure amount on an image sensor is inevitable by simply synchronizing the resetting timings of the electric charges of the electronic front curtain with the driving characteristic of the mechanical back curtain.
JP 2011-217368 discloses an image pickup method for reducing the uneven brightness in the captured image that could be otherwise caused by the electronic front curtain. The positional difference between the above electronic front curtain and the mechanical back curtain in the optical axis direction also causes other problems in addition to the uneven brightness. When the electronic front curtain and the mechanical back curtain are being driven, a partial (uneven) light shield occurs in which an F-number light flux is not shielded by the electronic front curtain but shielded by the mechanical back curtain. Thereby, an exposure time period of the image sensor by a ray on the electronic front curtain side in the F-number light flux becomes longer than an exposure time period of a ray on the mechanical back curtain side. This means that an optical transmission function (OTF) (or optical aberration) from the image pickup optical system to the image sensor changes in the image pickup (exposure) necessary for one captured image. Then, the image deterioration state also changes due to the change of the OTF in the image pickup.
Those methods disclosed in JPs 2012-175621 and 2011-217368 can restrain the rolling shutter distortion or reduce the uneven brightness in the captured image caused by the electronic front curtain. However, these methods cannot properly correct the image deterioration caused by the partial light shield by the mechanical back shutter when the electronic front curtain and the mechanical back curtain are used.